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Over the past 10 years, afterschool and youth develop¬ 
ment programming has moved from providing childcare 
for working parents to being an integral component of 
the learning day, supporting the academic, social, and 
emotional development of young people (C. S. Mott 
Foundation, 2007; Durlak & Weissberg, 2007). An im¬ 
portant part of that transition has been a growing em¬ 
phasis on improving program quality. Many communi¬ 
ties around the country have begun to create site-level 

continuous improvement models (Wilson-Ahlstrom & 
Yohalem, 2008; Yohalem & Wilson-Ahlstrom, 2009). 
Aligned performance measures help program adminis¬ 
trators evaluate the quality of young people’s experience 
and give them a framework for improvement. 

Many of these quality interventions target the lead¬ 
ers of afterschool organizations rather than simply di¬ 
recting attention to the teaching staff. Afterschool pro¬ 
gram managers often start their careers as front-line staff 


and work their way up to management positions with¬ 
out receiving training or education in how to lead an 
organization. They may not see themselves as instruc- 
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tional leaders and may not have training in how to 
change the direction and design of their organization or 
how to develop the people who work for them. 

The literature on school leadership and climate 
change highlights why a leader-focused approach makes 
sense. Researchers have demonstrated that improvements 
in school leadership can lead to improved teaching capac¬ 
ity and therefore to improved student achievement. In 
their meta-analysis of 70 studies of principal leadership, 
Waters, Marzano, and McNulty (2003) identified 21 sep¬ 
arate responsibilities of school instructional leaders, from 
fostering a culture of shared beliefs to establishing order 
to providing resources and professional development. 
Improvements in a leader’s ability to perform these re¬ 
sponsibilities were linked to im¬ 
proved student achievement. Other 
reviews of the research have simi¬ 
larly found that school leaders have 
a responsibility to set direction, de¬ 
velop people, and redesign the or¬ 
ganization in order to achieve im¬ 
proved student outcomes 
(Leithwood, Seashore Louis, 

Anderson, & Wahlstrom, 2004). 

Our study looks at how par¬ 
ticipation in a continuous quality 
improvement initiative produces 
higher-quality practice in Rhode Island’s afterschool 
community by fostering change in program management 
practices. Among other findings, we discovered that 
quality improvement begins with program managers, 
who then lead the process of change. 

The Rhode Island Program Quality Intervention 

The Youth Program Quality Intervention (YPQI), devel¬ 
oped by the David E Weikart Center for Youth Program 
Quality, is one intervention focused on program managers 
that is being used in communities across the country 
(Smith et al., 2012). YPQI is a multi-level intervention that 
uses continuous improvement practices to increase stu¬ 
dent exposure to positive youth development methods. 

In Rhode Island, development of a statewide quality 
improvement system based on YPQI began in 2004, 
when the Wallace Foundation awarded a large grant that 
allowed for the establishment of an afterschool interme¬ 
diary—the Providence After School Alliance (PASA)— 
and made quality an explicit priority. In partnership with 
the Weikart Center, PASA created the Rhode Island 
Program Quality Assessment (RIPQA), a tool comprising 
the Weikart Center’s validated Youth Program Quality 


Assessment (HighScope, 2005) and a locally developed 
administrative checklist. The RIPQA was piloted and 
rolled out statewide in 2006. Since then, PASA has part¬ 
nered with the Rhode Island After School Plus Alliance 
and the 21st Century Community Learning Centers ini¬ 
tiative at the state Department of Education to create an 
improvement system—the Rhode Island Program 
Quality Intervention (RIPQI)—with the assessment tool 
at its center. Close to 100 organizations across the state 
are engaging in the process, including all 65 of the 21st 
Century Community Learning Centers. The RIPQI in¬ 
cludes the following activities: 

• Training in the use of the RIPQA 

• Observation of individual program offerings at the 
point of service (where youth 
and adults interact) by teams of 
impartial external advisors and 
internal staff 

Assessment of management prac¬ 
tices including staffing and pro¬ 
fessional development supports, 
family and community engage¬ 
ment, and administrative practices 
Quality improvement planning 
with the support of a trained 
quality advisor 

Five hours of on-site technical 
assistance connected to the quality improvement plan 

• Participation in optional training aligned with the 
RIPQA 

Each participating organization is paired with an ex¬ 
pert “quality advisor” or coach for up to 25 hours every 
other year to complete the RIPQI process. First, a team of 
program staff uses the administrative checklist (RIPQA 
Form B) to rate the organization on various administra¬ 
tive practices. The advisor helps the team to arrive at con¬ 
sensus about strengths and areas for growth and to de¬ 
velop a quality improvement plan with specific action 
steps. Following this administrative audit, the advisor and 
the site director put together teams to observe three to 
five program offerings using the Weikart Center’s Youth 
Program Quality Assessment (RIPQA Form A). Again, the 
teams come to consensus and develop an action plan con¬ 
taining specific steps for improvement. Often these action 
steps include sending staff to PASA trainings to improve 
specific skills. The quality advisor participates in observa¬ 
tions, guides the site through this entire process, and then 
provides five hours of technical assistance or training in 
support of the site’s quality improvement action plan. 


Researchers have 
demonstrated that 
improvements in school 
leadership can lead to 
improved teaching 
capacity and therefore to 
improved student 
achievement. 
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Figure 1. RIPQI Theory of Change 


PASA has developed a theory of change that gov¬ 
erns this intervention, shown in Figure 1. The first box 
on the left represents the intervention itself and the ele¬ 
ments that comprise it. As the or¬ 
ganization begins to engage in the 
intervention, the program manager 
begins to make changes to his or 
her practice that in turn affect the 
whole organization. This im¬ 
provement leads to changes at the 
point of service. As instructional 
quality improves and youth are 
more engaged, we expect to see 
the improvement in youth out¬ 
comes the intervention was de¬ 
signed to produce. 

PASA is not the only organi¬ 
zation to create a quality improve¬ 
ment system based on the YPQI. 

In fact, to date, more than 70 
communities around the country 
are implementing all or some 
components of this model, 1 pro¬ 
viding substantial evidence of ef¬ 
fectiveness. For example, in Palm 
Beach County, Florida, the inter¬ 
mediary organization Prime Time 
Palm Beach County has been implementing a quality 
improvement system based on the YPQI for the past 
five years. A recent study of that model demonstrated 
that a quality improvement system centered around a 


valid assessment tool and associated coaching and tech¬ 
nical assistance can have positive effects on the quality 
of instructional and management practices in after¬ 
school programs (Sinisterra & 
Baker, 2010; Smith, Akiva, 
Blazevski, Pelle, & Devaney, 
2008). The Weikart Center, in a 
rare experimental study of a con¬ 
tinuous improvement interven¬ 
tion in an educational context, 
examined the effectiveness of the 
YPQI in 87 afterschool programs 
in five states. Results show that 
the YPQI had a substantial and 
statistically significant effect on 
both the continuous improvement 
practices of site managers and the 
instructional practice of front-line 
staff (Smith et al., 2012). Both 
studies provide critical context for 
understanding the likely effects of 
the RIPQI on manager and staff 
practice. Our study focuses on 
Iiow these effects occur, notably in 
the words of site managers en¬ 
gaged in the RIPQI process. 

Study Overview 

The goal for this study was to test the validity of the theory 
of change presented above, using two guiding research 
questions: 


The Weikart Center, in a 
rare experimental study of 
a continuous improvement 
intervention in an 
educational context, 
examined the effectiveness 
of the YPQI in 87 
afterschool programs in 
five states. Results show 
that the YPQI had a 
substantial and statistically 
significant effect on both 
the continuous 
improvement practices of 
site managers and the 
instructional practice of 
front-line staff. 
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1. Does the RIPQI process produce change in organiza¬ 
tions? 

• Is implementation of the RIPQI related to change in 
the quality of instruction and child engagement? 

• Is implementation of the RIPQI related to change in 
organizational context, administrative practices, and 
family engagement practices? 

2. How does change happen? 

• What practices do managers employ that may con¬ 
tribute to change at their site? 

• How do managers transfer, adapt, and extend the 
RIPQI in organizational settings? 

• In what ways are site managers affected by imple¬ 
mentation? 

The first set of questions focuses on the first three boxes 
in the theory of change: implementation of the intervention, 
change at the program level, and changes in instructional 
quality and youth engagement. The second set of questions 
explores what happens in the spaces between boxes to 
make change happen. The last step in the theory of change, 
the effect on youth outcomes, was beyond the scope of 
this study but is an important area for future research. 

To answer these questions, we used a mixed-methods 
approach, employing data from 53 afterschool programs 
across Rhode Island funded by the 21st Century 
Community Learning Centers office at the state 
Department of Education. Sites are required to partici¬ 
pate in the RIPQI process every other year; at the time of 
the study, every site had participated at least once. The 
53 sites are distributed across the state, with a large con¬ 
centration in Rhode Island’s five “core” cities—Central 
Falls, Pawtucket, Newport, Providence, and Woonsocket. 
The sites serve all age groups with about half (54 percent) 
serving elementary-age students and the remainder serv¬ 
ing middle (30 percent) and high school (16 percent) stu¬ 
dents. Sites range in size from 15 to 200 students per day. 

Data Sources 

Our study uses the following sources of data: 

• Existing instructional quality data collected by trained 
observers during 2007-2010 (n = 325 program obser¬ 
vations) 

• Surveys with program staff (n = 62) and managers 
(n = 29) 

• In-depth interviews with a subset of managers (n = 6) 
who reported a high level of RIPQI implementation 

Observations were conducted using the Weikart 
Centers instrument (for validation evidence see Smith & 


Hohmann, 2005), one component of the RIPQA. 
Observations were conducted during individual program 
offerings over multiple sessions involving the same staff, 
the same youth, and the same purpose. Each required at 
least 45 minutes of observation by a reliable rater. The 
tool measures instructor practice in four key domains: 
safe environment, supportive environment, interaction, 
and engagement. Each domain has several indicators. 
Instructors are rated on a three-point scale using a rubric. 

Two surveys were used for this study, one designed 
for managers, including site coordinators and other ad¬ 
ministrators, and one designed for front-line staff who 
work directly with youth. The surveys were modeled af¬ 
ter those used in the YPQI study, described above, in an 
effort to create items and subscales in line with the 
known reliability and validity of those tools. The re¬ 
sponse rate was about 40 percent for the manager survey, 
with 29 managers, representing 21 of the 53 sites, re¬ 
sponding. The staff survey had a 26 percent response 
rate; the 33 staff members who responded represented 
14 out of the 53 sites. 

Finally, the interviews were conducted using a stan¬ 
dardized open-ended approach. Lasting about one hour, 
they included 15 questions in four key domains: changes 
to practice, accountability for implementing change, 
changes to leadership style, and overall program improve¬ 
ment. Five of the six managers interviewed were chosen 
because they reported high levels of RIPQI implementa¬ 
tion on the survey. We also attempted to use the survey to 
identify a low-implementing manager. Only one individ¬ 
ual had low enough scores to merit consideration as a 
contrast to the others, but her interview revealed that she 
reported low levels of implementation and change be¬ 
cause she had recently completed the process and had not 
yet conducted extension activities or seen change happen 
at her site. We therefore simply included this manager’s 
feedback with that of the other five. 

Data Analysis 

To analyze the data, we first looked at each data source 
individually and then began to link sources to answer the 
two research questions. Beginning with observational 
data, we identified 13 sites with data for two individual 
program offerings at each of two time points in different 
program years. We aggregated each site’s ratings for each 
time point and then compared the two time points to de¬ 
scribe an average amount of change for each site. Next we 
identified 21 instructors from different sites who had ob¬ 
servational ratings in different years and then compared 
the two time points to describe an average amount of 
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change for each individual. For each of these samples, the 
R1PQ1 had been carried out between the two time points. 

For survey analysis, we created several subscales from 
the survey items and ran cross-item and cross-survey anal¬ 
ysis to understand the characteristics of individuals and 
organizations who reported high and low levels of change 
in program practices and higher and lower levels of youth 
engagement as a result of the RIPQ1 process. 

Finally, to analyze the interview data, we first read 
through each interview transcript, looking for any practice 
or performance changes reported by the site managers we 
interviewed. We then identified key themes that emerged 
across all the interviews. 

Does Change Happen? 

Analysis of the observational as¬ 
sessment data, combined with staff 
and manager reports on the sur¬ 
veys and interviews, suggest that 
the R1PQI is working. To begin 
with, we saw high levels of fidelity 
to the RIPQI across all sites, in part 
because many of its elements are 
required. We quantified the level of 
implementation by asking managers 
and staff about their participation in various elements of 
the intervention, such as attending training, conducting 
observations, assessing administrative practices, and en¬ 
gaging in quality improvement planning. Out of a total of 
21 elements, nine required and 12 not required, the av¬ 
erage number in which managers participated was 10.5. 
More than half (55 percent) participated in 11 or more 
elements. We further distinguished high implementers 
from low implementers by looking at the 12 optional or 
“extension” activities, which required additional effort on 
the part of managers and staff. On average, managers par¬ 
ticipated in 6.75 of these extension activities. 

Youth program staff across Rhode Island reported 
that administrative practices and instructional experiences 
are improving as a result of the RIPQI. Managers and staff 
reported almost universally (97 percent of managers and 
81 percent of staff) that the RIPQI produced positive 
change in program quality. Fully 72 percent of managers 
and 67 percent of staff reported that the RIPQI supported 
youth to become more engaged in program offerings. 

In analyzing the observational data, we looked at the 
subsamples of 13 sites and 21 individual instructors from 
different sites who had observational data before and af¬ 
ter participation in the RIPQI. We asked the simple ques¬ 
tion: Was there positive change from the first observation 


to the time after the RIPQI had been introduced? On the 
whole, the answer was yes, although the small sample 
size reduced the power to detect statistically significant 
differences. In nearly all cases, scores went up from the 
baseline to the second observation. For the subsample of 
13 sites on which we had instructional quality data at 
two time points, differences in observed quality were 
positive, particularly in the domain of safe environment, 
where we saw statistically significant change. 

Our best test of baseline-to-post-RIPQI change is 
for the 21 individual instructors who were observed 
doing the same program at two time points, with 
exposure to the RIPQI in between. 
In these cases, the average score 
change was large and statistically 
significant. While scores improved 
in all four key domains, statistically 
significant change occurred in the 
total score as well as in two 
domains: supportive environment 
and interaction. 

How Does Change Happen? 

That the RIPQI is working was 
one question this study set out to 
answer. Our findings provide evidence supporting the 
theory of change. When fully implemented—that is, 
when staff embrace the process and engage in activities 
beyond what is simply required—the RIPQI does appear 
to produce measurable change in instructional practice. 
These findings serve to confirm with local data what 
the more rigorous studies described above suggest: 
that the investment in the RIPQI has produced gains 
in the quality of afterschool programs across the state. 
However, perhaps the more intriguing finding from this 
study is how the RIPQI is working. If we understand the 
how, we can improve training for sites and better prepare 
quality coaches. 

The manager interviews allowed us to further ex¬ 
plore how administrative practices support changes to 
instruction. Across the interviews, several themes 
emerged regarding how the RIPQI changed management 
practices and policies. 

Changes to Manager Practice 

Managers reported changes in how they viewed or car¬ 
ried out their roles. Several talked about being more 
comfortable in the role of instructional leader, being 
more able to provide feedback to program instructors, 
freeing up time to provide better supervision, and in gen- 


Youth program staff 
across Rhode Island 
reported that 
administrative practices 
and instructional 
experiences are 
improving as a result of 
the RIPQI. 
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eral being more intentional about how they ran their 
sites. For example, one manager reported: 

I used to just think that I’d hire the facilitators and 
they’d know what to do and how to interact with 
kids, or teachers would come on board and it’d be, 
“Oh, they’re a school teacher so I don’t really have to 
tell them anything,” but.. .1 think I’m more comfort¬ 
able speaking up to facilitators now, and I do it more 
often.... I think that I [have] become a stronger lead¬ 
er because I’m more intentional about the supervi¬ 
sion and the feedback that I 
give people in the observa¬ 
tions that I do with them. 

Another core part of changing 
the manager role was improving ori¬ 
entation and training for staff. All six 
managers mentioned this element 
and described how incorporating the 
RIPQA into their training allowed 
them to better prepare their staff to 
meet expectations. One manager ex¬ 
plained that the RIPQA “gave me 
some way to structure my trainings with my staff, and it gave 
them a structure of how to think about their time in the class¬ 
room with students.” Another talked about how she selected 
one indicator from the RIPQA to discuss at each staff meeting. 

Improvements to Communication 

One of the most prevalent themes that came out of the 
interviews was improvement to communication at all 
levels. Managers reported better communication between 
site managers and staff, site managers and their supervi¬ 
sors, the program and parents, and instructors and youth. 
Communication was defined broadly but included some 
of the following types of changes or improvements: 

• Improved policies and procedures, clarifying to every¬ 
one what was expected of participating youth and staff 

• More intentional and more regular supervision of staff, 
including not only formal supervision but also more 
informal observations, check-ins, and meetings 

• Improved communication with youth, including more 
opportunities for youth to voice their opinions and 
have a say in the program structure through, for ex¬ 
ample, youth advisory councils, student focus groups, 
and one-on-one conversations with instructors 

• Better staff meetings that took advantage of the RIPQA 
tool and its core indicators 

• Improved staff connections to family and community, 
more parental involvement 


One manager talked about a change she made to the 
program schedule to allow for more communication 
among staff: 

Every day, as an entire group, we meet at 2:00—ev¬ 
ery single day. And we have a check-in about the 
day, talk a little bit about the logistics of the after¬ 
noon. . .and then, there’s that open hour.. .where ev¬ 
eryone is paid to be at work to.. .do lesson planning, 
get their snacks ready, meet with each other...or 
someone will have scheduled a time to come and 
meet with me one-on-one. But we 
have that hour set aside every day. 

Another talked about how she 
uses the RIPQA to help herself and 
her team set goals: 

During my supervision that 1 
have with them on a monthly 
basis, that’s one of the talking 
points that we have, is the 
RIPQA process. And how do 1 
feel that it’s going, are there 
any issues going on, is there 
anything that I think we can strive [for].... 
I’m constantly setting goals every month, talking 
about the goals that we’ve met. 

Improvements in Program Structure 

Several managers reported that the RIPQA process pro¬ 
vided them with a practice that helps them shape and 
make improvements to the overall program structure. As 
one manager put it, the RIPQA “creates procedure. It cre¬ 
ates form. It creates a structure that you can work within 
that is still flexible.” Another reported: 

1 think the biggest thing is just offering a structure 
that seems to really work...There’s just so much 
when you’re trying to hire, and staff, and train youth 
workers coming from such different backgrounds. I 
cannot assume that they’re coming in with a certain 
skill set, and [the RIPQA] has kind of allowed me to 
structure our programming and structure the way 
we think about how we interact with students. 

The changes to program structure that resulted 
from the process were not purely abstract. One manager 
talked more concretely about how the process helped 
her restructure her program: 

One of the main things that came out of our RIPQA 
process two years ago was that we were working our 
kindergarten and first graders way too hard.... We 


One manager explained 
that the RIPQA "gave me 
some way to structure my 
trainings with my staff, 
and it gave them a 
structure of how to think 
about their time in the 
classroom with students." 
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restructured our K and 1 program based on the 
RIPQA process.... And there was a huge change in 
behavior, and meltdowns, and kids passing out at 
4:30—-just falling asleep because they were so ex¬ 
hausted because we worked them so hard. 

Improvements in Hiring Practices and 
Staff Composition 

A fourth theme that came out of the interviews was the 
impact the RIPQA process had on the composition of the 
staff. All interviewed managers talked about such efforts 
as rewriting job descriptions to better reflect the quality 
standards, making changes to the organizational chart to 
allow for better staffing, creating assistant director posi¬ 
tions in order to free up the site manager to spend more 
time on quality improvement, and firing staff or using 
natural transition to eliminate staff who were not com¬ 
mitted to reflective practice and improvement. One man¬ 
ager described this last kind of change: 

After about a year of RIPQA, when I realized that 
there were some staff that were either apprehensive 
or completely just holding back from being a part of 
this and moving forward like we were, they were not 
asked to come back to work this year... They may be 
wonderful youth workers, but if they’re not aligned 
with the vision and the needs that your school has, 
then it’s just not the right fit anymore. 

Improvements in Instruction 

Of course the changes listed above are valuable only if 
they eventually have a direct impact on the experience 
of young people in the program. Although at least two 
of the managers felt that the changes they were making 
had not yet led to improved instruction, others talked 
about what they saw changing for the young people in 
their programs, including: 

• Improvements to the safety and environment includ¬ 
ing more secure entrances and sign-out procedures, 
more appropriately sized furniture, and better fire drill 
procedures. 

• Improvements to the quality of interaction between 
the youth and the staff. For example, staff asked more 
open-ended questions and were more intentional 
about greeting each student; youth voice was solicited 
through time built in for feedback and reflection. 

One manager reported on how the program elicited 
youth voice: 

The ten-minute, five-minute check-in at the end of a 
class, “How did this go for you? What’s your favorite 


part? How can we make it better?” —that was some¬ 
thing that kids really did come to me and say, “Hey, 
guess what? We told them we didn’t like this class 
this day and they’re going to change it.” And that 
was a big thing. 

The How of Program Improvement 

These findings describe changes to administrative and 
management practices that can lead to improved instruc¬ 
tion and increased youth engagement as described in the 
theory of change. When fully implemented, the RIPQI 
does appear to produce significant change in instruction¬ 
al practice, as measured by the observations, as well as in 
greater youth engagement, as reported by managers and 
staff. In addition, it appears to have an effect on manage¬ 
ment practice, as described by the program managers 
interviewed for this study. 

So what is actually going on at the site level that 
makes change happen? Taken together, the observational 
data, survey responses, and interview transcripts begin 
to tell a story that mirrors the theory of change laid out 
above. That is, sites appear to go through a flow of activ¬ 
ity that starts with structural change and ends with im¬ 
provements to instructor practice: 

Structural change: 

administrative practices, hiring and firing, policies 

Organizational and climate change: 
communication, training 



Manager-level change: 
becoming instructional leaders 



Changes to instructional practice: 
youth experience in the program 

Structural Change 

It appears that change begins at the higher levels of 
administration. The RIPQI provides a framework and 
context for getting the right staff in place to do the 
right jobs. By revising job descriptions, hiring more in¬ 
tentionally, firing staff who aren’t a good fit, creating 
new policies and procedures, and shifting job duties, 
organizations ensure that their staff members are strong 
and committed and that they understand exactly what 
is expected of them. 
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Organizational and Climate Change 

Once the right mix of staff is in place, the RIPQ1 seems to 
provide the staff with a framework for improved com¬ 
munication. This communication takes several forms, 
but the most common changes seem to be dramatic im¬ 
provements to orientation and training for new and re¬ 
turning staff and more intentional staff meetings. By us¬ 
ing the RIPQA to shape new orientations, managers 
ensure that all staff members have a set of shared stan¬ 
dards to work from and can therefore establish common 
goals. After establishing a common language at the be¬ 
ginning of the year, managers then 
used aspects of the RIPQA through¬ 
out the year at staff meetings. The 
standards provided managers with 
a structure around which to shape 
meetings intentionally. 

The survey data tell us that 
staff who are most likely to report 
changes in their practice are those 
who feel most supported by their 
supervisors and who feel they un¬ 
derstand the shared goals of their 
organization. It stands to reason, 
then, that, as the climate of the or¬ 
ganization becomes more inten¬ 
tionally aligned with the RIPQA and staff are receiving 
more training and better support through ongoing super¬ 
vision and staff meetings, they will feel more supported 
and therefore more inclined to enact change. 

Manager-Level Change 

As managers become more certain of their staffing mix, 
create a shared language for the staff, establish clearer 
policies and procedures, and develop an infrastructure 
for intentional staff meetings, they begin to feel more 
confident as instructional leaders. Every manager talked 
about continuing to conduct informal and formal obser¬ 
vations after the official RIPQI process was over. These 
managers now have language for giving staff feedback on 
their performance. Many also talked about establishing 
more regular and intentional supervision with their staff, 
using the RIPQA as a guideline. When staff are hired and 
trained using a common language, managers can more 
easily provide guided support for their practice. The sur¬ 
vey data suggest that the front-line staff most likely to 
change their practice are those who are involved deeply 
in the quality improvement process. As managers be¬ 
come more comfortable giving feedback, they are likely 
not only to observe their staff, but also to provide recom¬ 


mendations and feedback that lead to the final product: 
improvements to instructional practice. 

Changes to Instructional Practice 

The final stage in the theory of change that our study ad¬ 
dressed is improvement to instructional practice. This 
process of change—improving the staffing mix; creating 
a shared language and common goals; and more inten¬ 
tionally supporting staff through improved communica¬ 
tion, training, and supervision—takes time. Sites that 
have engaged in the process longer or that have strong 
leaders are further along than oth¬ 
ers. Several managers, but not all, 
did report change at the instructor 
level. Many of the changes manag¬ 
ers described were basic and rela¬ 
tively easy to achieve, such as 
greeting all youth warmly, improv¬ 
ing the appropriateness of furni¬ 
ture and supplies, and creating a 
sense of belonging. However, a few 
managers referred to development 
of higher-order skills among their 
instructors, such as asking more 
open-ended questions, providing 
opportunities for youth to reflect 
on the program, and doing more intentional planning. 

Study Limitations 

This study has several important limitations. For one, it 
used existing, but incomplete, observational data col¬ 
lected as part of a quality improvement system. Not ev¬ 
ery site had a complete set of observational data at two 
time points. We based our analysis on those that did. 

A second limitation is the small sample size. As noted 
above, we had a relatively low response rate on the staff 
and manager surveys, probably because we distributed 
them in June, when many programs were breaking for the 
summer. By design, interviews were conducted with just 
six individuals. With more time and better response rates, 
the data might have yielded different findings. 

A final limitation is that the study was conducted by 
someone very close to the RIPQI process. Elizabeth 
Devaney created the RIPQI in partnership with the 
Weikart Center and has been largely responsible for its 
growth and development into a quality improvement sys¬ 
tem in Rhode Island. She is not an impartial researcher. 
Those surveyed and interviewed knew Elizabeth well and 
may have tailored their responses to her. However, her 
closeness to the sites was also a benefit because she was 


As managers become 
more comfortable giving 
feedback, they are likely 
not only to observe their 
staff, but also to provide 
recommendations 
and feedback that lead to 
the final product: 
improvements to 
instructional practice. 
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intimately familiar with the RIPQI and the nuances of 
implementation. 

Further research is needed to confirm the validity 
of these findings and to explore what effect additional 
factors, such as the experience level and education of 
the manager, the longevity of staff, and the program set¬ 
ting, may have. Although these findings mirror the edu¬ 
cation literature on administrator effect on teacher 
practice (Leithwood et al., 2004; Marzano et al., 2003), 
there may be other ways to understand the flow of fac¬ 
tors that affect instructor improvement, including the 
effect of formal education and training. The field would 
benefit from additional research 
exploring the pathways to instructor 
improvement. 

Implications 

The purpose of the study was to 
gain a better understanding of how 
the RIPQI achieves effects on after¬ 
school organizations in Rhode 
Island. Its findings have implica¬ 
tions both locally and nationally. 

Locally, these findings suggest that 
Rhode Island’s quality improve¬ 
ment system is working but is 
highly dependent on administra¬ 
tors embedding the process and the language of the 
RIPQA into their organizations. Managers who can trans¬ 
late a one-time assessment and quality improvement pro¬ 
cess into an ongoing, embedded system of continuous 
improvement are going to be more successful than those 
who can’t. Knowing that, RIPQI decision makers may 
want to redesign training for new sites and quality coach¬ 
es to include strategies for embedding the process into 
ongoing program planning. For example, bringing suc¬ 
cessful managers into the training to share lessons 
learned and promising strategies may improve imple¬ 
mentation at new sites. 

Nationally, this study can inform communities that 
are developing and launching quality improvement sys¬ 
tems based on the same or similar tools and practices. A 
clear lesson from this study is that focusing on managers 
at the start may be more effective than moving directly 
to individual instructors. Without a shared language 
and infrastructure for discussing quality improvement, 
instructor-level change may not happen or may be 
short-lived at best. Change seems to happen on a con¬ 
tinuum that begins with the administration. 
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Note 

‘Use of other continuous improvement models is also 
growing. Notably, the Afterschool Program Assessment 
System (APAS) by the National Institute on Out-of- 
School Time (NIOST), which is being used in several 
communities around the country, draws on similar 
continuous improvement practices and principles. 
Validation evidence for the APAS is provided in the final 
report on the Massachusetts Afterschool Research Study 
(INCRE & NIOST, 2005). 
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